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In this work we report new results relating to decision feedback equaliza- 
tion. The equalizer and the transmitting filter are optimized in a PAM 
data communication system operating over a linear noisy channel. We 
use a mean-square error criterion and impose an average power con- 
straint at the transmitter. Assuming correct past decisions, an explicit 
formula for the minimum attainable mean-square error is given. The 
possible advantages of signaling faster than the Nyquist rate while de- 
creasing the number of levels to maintain the same information rate are 
investigated. It is shown that, in all cases of practical interest, signaling 
faster than the Nyquist rate, while keeping fixed the information rate, 
increases the mean-square error. Finally, to illustrate the use of the 
results, application is made to a cable channel where the loss in dB varies 
as the square root of frequency. Various asymptotic formulas and curves 
are provided to exhibit the relationships between the quantities of interest. 

I. INTRODUCTION 

A great deal of research, particularly in the past decade, has been 
expended on the problem of linear equalization. This has yielded a 
considerable body of theory and technology making possible the design 
of apparatus for successfully combating intersymbol interference in 
PAM data transmission systems operating over noisy linear channels 
where delay distortion predominates. Since linear equalizers must com- 
pensate for the channel characteristics in the presence of noise, they 
cannot be expected to perform well over severely frequency-attenuating 
channels or channels possessing nulls in the amplitude characteristic. 

Interest in the high data rates over voiceband and cable channels 
inevitably leads to the search for more effective equalization methods. 
Faster pulse rates place signal energy well within the badly attenuated 
portion of the transmission spectrum, resulting in severe intersymbol 
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interference correctable by linear methods only at the expense of a 
significant enhancement of the noise. 

A "bootstrap" technique, commonly referred to as "decision feed- 
back," when combined with linear equalization can yield significant 
performance improvement. 1 - 2 In this method the samples of the pulse 
tails (postcursors) interfering with subsequent or future data symbols 
are subtracted without incurring a significant noise penalty. The effect 
of pulse tails (precursors) which occur prior to detection and inter- 
fere with past symbols is minimized by a conventional linear equalizer. 

Much has been written about this subject. In a fundamental paper 
where an excellent bibliography can be found, Robert Price 3 demon- 
strated quantitatively the merits of decision feedback equalization in 
certain applications. 

In this work we jointly optimize the receiving and transmitting 
filters in a PAM data transmission system employing decision feed- 
back. The chief difference between our work and Price's is in the choice 
of performance criterion. We minimize mean-square error while 
Price maximized the signal-to-noise ratio under the constraint that the 
overall intersymbol interference be zero. Our criterion is not as strin- 
gent as Price's and allows trade-offs between added noise and inter- 
symbol interference. Monsen 4 also investigated some aspects of our 
problem but did not arrive at a complete solution. Our chief contribu- 
tion is an explicit formula for the minimum mean-square error (MSE). 
The simplicity of the formula makes possible detailed investigation of 
optimized system performance. 

In Section II the model is stated and the problem is formulated. 
In Section III the receiving filter is optimized and in Section IV the 
transmitter filter is optimized. In Section V we examine the problem 
of signaling faster than the Nyquist rate and finally in Section VI 
we use our results to investigate in detail the performance of a data 
system operating over a cable channel. 

II. THE MODEL AND PROBLEM FORMULATION 

The system model under investigation is depicted in Fig. 1. The data 
signal denoted by D(t) is passed through the transmitting filter having 
an impulse response s(t) and giving rise to an average transmitted 
power P. The data symbols {a n }", are independently picked at the 
rate l/T and take on values with equal probability from the set 
[=fc 1, ± 3 ± 5 • • • ± (L - 1)} where L is an even integer. The 
resulting signal is admitted to a linear channel characterized by an 
impulse response h(t). The received signal plus noise is processed by 
the equalizer which is comprised of a linear filter having impulse 
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Fig. 1 — Block diagram of the model. 

response w(t), a sampler, a decision rule, and a feedback digital filter 
characterized by the infinite set of real numbers {&»}". The added 
noise n(t) is a zero-mean white random process with double-sided 
spectral density N /2. The output data symbols are denoted by d„. 
The general problem we would like to solve is the minimization of 

MSE = E{v k - d k } 2 

with respect to the set of square integrable functions {s(t), w(t)} and 
the infinite sequence of numbers [b n \. This is to be carried out when 
the channel impulse response h(t), transmitted power P, and a decision 
rule are given. The symbol E(-) denotes expectation with respect to 
all the random variables. 

The nonlinear relation between the estimated symbols {d n \ and the 
input symbols {a„j makes this problem mathematically intractable. 
However, by assuming that past decisions have been correct, we can 
begin to approach the problem. The resulting MSE must then be 
interpreted as a lower bound on the true MSE. Alternatively, if no 
errors have occurred in the past, the MSE under this assumption 
provides an indication of the noise immunity of the system (including 
residual intersymbol interference). 

Let r(t) = s(t)*h(t)*w(t) denote the overall impulse response, 
where * denotes convolution. Under the assumption of correct past 
decisions, the received sample taken at time t = kT is 



v k = L r n a k -n - E b n a k - n + n(t)*w(t)\ t . 
and the mean-square error is then by definition 
MSE = E J £ r n a k - n + £ (r„ - b n )a k . n 

n= — » n = l 



■ kT 



+ n(t)*w(t) | t = kT + (r - l)a* 
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A straightforward calculation gives 

MSE = <r 2 £ rl + o*{n - D 2 + o*£ (r» - &») 2 + <r 2 , 
where 

and 



= \° f^(t)dt. 



It can be immediately concluded that the mean-square error is 
minimized by setting b n = r n , n = 1, 2, ■ • • , which eliminates the 
feedback coefficients {&„} from further consideration. 

The problem we now confront is the dual minimization of 

MSE[s(0, w(fl] = d T £ r* - 2r +1+41 

|_ n = - oo ff aJ 

with respect to s(0 and «>(<) when a constraint is imposed on the 
average transmitted power. 

The above expression indicates that, under the assumption of 
perfect past decisions, the mean-square error is minimized by minimiz- 
ing both the pulse precursors in the overall impulse response and the 
output noise power, while keeping r close to unity. 

We give a precise formulation and solution to this problem in 
Section III. 

III. RECEIVER OPTIMIZATION 

Writing the MSE in detail we obtain 
MSE _ ° r '- " |2 

9 1 I 



£ [ r w(r)p(nT - r)drT 

n= — « |_ J— co J 

- 2 r w(r)p(- r)dr + ^ 2 f V wKr)dr, (1) 



N 
- 2 / w{r)p{- T)dT ■+■ 

7—00 

where 



p(0 = s(t)*h(t). 

Keeping s(t) fixed, and using a standard calculus-of- variation ap- 
proach, results in an integral equation for w(t) 

p(- t) = iVo^CO + f M w ^ ( £ P( nr ~ r) P (wT " f) ) dr ' (2) 

7—00 \ 71= — » / 
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where 



If in cq. (2) we set 



U„ = J" w(r)p(nT - r)dr, 



we see that the optimum solution must have a representation in the 
form 



where? 



and 



w(t) = £ g„p(nT - t), (3) 

n— — x 



00 = JT>- (1 — ^o) 



L n - . 

5,= -^, n=S-l. 

In (3) is revealed the structure of the optimum receiving filter. It is 
composed of a matched filter having impulse response p(— t) followed by 
a one-sided (anticausal) lapped delay line with weights equal to g n . 

Linear equations involving the set JC/„) can be obtained by first 
multiplying both sides of (2) by p(kT — I), k ^ 0, then integrating 
from minus to plus infinity. The resulting linear system of equations is 

R k = N' U k + E R n - k U„, fc-0, -l---, (4) 

«=— -j-. 

where 

R k = R- k = J2 p(- t)p(kT - t)dt. 

The system of eqs. (4) can be solved by standard Wiener-Hopf tech- 
niques and the details are given in Appendix A. The solution in terms 
of the discrete Fourier transform of the sequence | L 7 „J°_ M is 

Vie) = £ U n e inB 

= l - W%V > (5) 
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where 

M(d) = M+(0)M-(8) = R(d) +N' = "if M n e ine , 



M+(6) = L y*e M , 



-0 



and 



M-(e) = M+(- e). 

This is standard procedure making use of the well-known factoriza- 
tion property of covariance functions. Methods for obtaining the 
sequence {Yn}<T from the given sequence {#„}-. are well documented 
in the literature. One method is summarized in Appendix A. 

Having specified the optimum receiving filter, we now obtain a 
formula for the minimized mean-square error. The availability of this 
simple formula will allow further optimization of the transmitting 
filter. 

Let w (t) be the impulse response of the optimum receiving filter. 
[This function solves the integral equation (2).] Substitute w (t) 
into (2), multiply both sides by w (t), and integrate from minus in- 
finity to plus infinity to obtain 

r p(- t)w (t)dt = N' I wl(t)dt 

J— oo J—m 

+ Z ( T v(nT - t)wo(t)dtY- (6) 

n = — " \ J— oo / 

Putting this into (1) with w(t) replaced by w (t) we get a formula for 
the optimized MSE 

MSEOo(0] = <r 2 a {\ - Uo) = N' a 2 a g . (7) 

This result was obtained by Monsen 4 but unfortunately he did not 
go any further. As it turns out, a much richer formula than (7) can be 
obtained since U can be expressed directly in terms of the spectrum 
of the channel characteristics in cascade with the transmitting filter. 
To carry this further, observe from (5) that 

Uo = dc term in U(9) 



= 1 - 5 (8) 



and consequently 



75 



MBE-*Snf- (9) 

To 
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As it turns out, 70 is functionally related to M{0) in a rather simple 
manner. This relationship can be found in the literature but the deriva- 
tion is short and so we briefly outline the approach. 

Under very mild conditions on M(6) (see Doob, 6 pp. 159-161) 



E Tne'" 9 

n=0 



M(d) = 

where 70 is real and positive. Since M{6) > 0, consider 



(10) 



f* In M(6)dt = f W In [7c 



+ E 7«e*' 



+ 



/.: 



In 



dd 



70 + E y n e- inl 

n = l 



dd. (11) 



When the ln's on the r.h.s. of (11) are expanded in a power series 
and the integrations are carried out (recognizing that all integrals 
involving powers of exp { ind } , n 7* 0, vanish) we get 



where 



yl = exp I i /_'. lnLR{9) + N '°1 de }> (12 ^ 



R(9) = £ R n e ine , 



.-/."'wiv-s. 



Rn — 



and 



u) = I s(t)*h(t)e iut dt. 



P(co) = 



After minor algebraic manipulations and changes of variables we 
obtain by substituting (12) into (9) 



MSE = ff 2 a exp 



T fir I 

-± / ln[F(«) + l]d« , 



where 



'm-^XI^-t) 



(13) 



(14) 



This formula, as far as can be determined, is new and its simple 
form will enable us in Section IV to carry out an additional optimiza- 
tion with respect to the transmitting filter. 

It is instructive at this point to compare this formula with the one 
obtained for a linear equalizer without decision feedback. Berger and 
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Tufts 6 have found such a formula and from their paper we have that 

rp rrIT 



(MSE)ii..., = <rlf- r [F(«) + l]-»d« 

to I—kit 

"\F(«) + 1/' 



(15) 



where 



rp rx/T 
^7T J- r /T 

In terms of the same notation, (13) can be put into the form 

MSE = exp{ -<ln[y(«) + 1])) (16) 

from which we get immediately that 

MSE g /<r-i»ir<.>+i]\ = / ^J . (MSE) linea , (17) 

As expected, the mean-square error with decision feedback is always 
smaller than the MSE of a linear equalizer. Comparing (15) with (16) 
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Fig. 2— Amplitude-squared characteristic of typical voiceband channel. 
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Fig. 3 — MSE in dB vs binary data rate for channel shown in Fig. 2 without dc 
transmission. 



shows that both equalization methods yield the same MSE if and only 
if F(co) is a constant, i.e., there is no intersymbol interference. 

Prior to optimizing the transmitting filter we wish to illustrate the 
behavior of (15) and (16) for a typical voiceband channel as the signal- 
ing rate \/T is allowed to increase. An amplitude-squared character- 
istic for a typical voiceband telephone channel is shown in Fig. 2 
(the dashed line with zero transmission at zero frequency is typical). 
Figure 3 shows the resulting MSE vs pulse rate for both a linear equal- 
izer and a decision feedback equalizer when a\ = 1 (binary data) and 
— 10 log N' p^ 40 dB. The calculations were done numerically by 
using (15) and (16). We note that the performance of the linear 
equalizer deteriorates rapidly when the rate is greater than « 3000 
bits/second while the decision feedback equalizer deteriorates grace- 
fully. The reason the linear equalizer shows such a poor performance 
is that it must compensate for the missing energy around dc. In 
practice, of course, modulation is used to place the data energy at a 
more suitable location in the passband spectrum to avoid this severe 
null. To make the comparison fair, we artificially extended the channel 
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Fig. 4 — MSE in dB vs binary data rate for channel shown in Fig. 2 with dc 
transmission. 

characteristic from about 300 Hz to Hz to a constant transmission. 
This is indicated in Fig. 2 by the dashed line parallel to the frequency 
axis. Figure 4 shows the comparisons for this atypical channel. As 
expected, the linear equalizer has a sharp threshold at approximately 
the Nyquist rate but, as before, the decision feedback equalizer 
deteriorates much more gracefully. 

IV. TRANSMITTER OPTIMIZATION 

The problem we address here is the optimization of (13) with 
respect to the transmitting filter characteristics subject to an average 
power constraint. 

Let S(<a) and #(«) be the Fourier transforms of s(t) and h(t) re- 
spectively. The average power at the output of s(t) is 

p = t />>«< - a £ *•>*• 

where by S 2 (u>) we mean \S(a) | 2 . 
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The problem at hand is to maximize the functional 

J-r/T n 

+ x r /r CE52(«)]&o (19) 

J-r/T n 

with respect to the infinite set of functions \Sl(u>), n = all integers). 
Where 

2 



Sl(u) = \S ( ai ^- j 



X = 1/AToT 1 



and X is a Lagrange multiplier to be determined from the constraint 
on the average transmitted power. Observe that these functions 
are independent over the range — t/T ^ u ^ t/T and therefore 
consider the variation of I with respect to, say, S 2 j(a)) 

s < T = C I k s gfgfe + 1 8S?M + m ' M \ d -- (20) 



When S 2 j(co) ^ 0, setting (20) to zero implies 

Kfflfa) 



K Z S*( u )H*(w) + 



+ X = (21) 



for all w G [-*/r, t/T]. Now, assume that H*(a) > Hl(a) for 
1 72. | < \m\ and a> £ [—t/T, jt/T]. When this condition is satisfied it 
is not possible to solve the system of equations given in (21) unless 

Sg(w) = for all n * j 

in which case we get 

KH 2 j(u) = - X[JfcS5(M)ffJ(«) + 1]. (22) 

Substituting this into (20) indicates that the largest value is obtained 
when Sj(u) = 5q(w) = S 2 (u) and furthermore X must be negative. 
So far we can conclude that for channels possessing monotonically 
decreasing amplitude characteristics, i.e., Hfnioi) > #»(<*>), — t/T 
^ a) ^ t/T, when |n|>|ra|, the optimum transmitting filter cuts 
off at the Nyquist frequency t/T. The optimum system allows no 
transmission outside the band |o>| > t/T. The restrictions imposed on 
the channels are mild and are expected to be satisfied in most situa- 
tions of interest. However, removing these restrictions makes the 
problem slightly more complicated, and it is left up to the reader to 
reason how it can be solved. 
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Making use of this partial solution permits writing the mean- 
square error in the simplified form 



- In 



/ MSE \ T r' T ln LKH 2 (u)S 2 (u) + 1]dM (23) 

\ (Ta / 41T J-ir/T 

and the functional now to be further maximized with respect to the 
inband structure of S 2 (io) reduces to 

ir8(a)l = r /T ln [KS 2 (o>)H 2 {a>) + l]dco + X V S 2 («)d«. (24) 

J-ir/T J—x/T 

As can be seen from (20) and (22), eq. (24) is maximized when 



S 2 (co) = max 



KH 2 (o>) - Xo n 



(25) 



where X = — X . 

To determine the Lagrange multiplier X , two cases must be dis- 
tinguished. 

Case 1: KH 2 {u) - X„ > 0, for all \<a\ ^ t/T 

In this case, we get 

1 + KH 2 S 2 = y.H 2 , ix = K/X , 

which when substituted into (23) results in an explicit expression for 
the optimum MSE 

_ i n ( M|E ) = l n M + I ["* In H»(«)d». (26) 

The factor y. is determined from the average power constraint in the 
following manner. Use (25) and (18) to write 

- Si (m - A), (27) 



where 

a- if'" ' 

r Jo 



H 2 (u) 



do. 



Substituting the parameters K = 1/TNq and Nq = N Q /2a 2 a into (27) 
gives explicitly 

ti = P + A, (28) 
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where 

P Average transmitted signal power 

/No 1 \ ~ Average noise power in the Nyquist band 
V 2 Tj 

Thus (2G) and (28) provide a complete solution for this case. 

Case 2: There exists a set of co for which KH 2 (<x>) — X ^ 

The optimization procedure in this case involves the standard water- 
pouring argument. To illustrate the nature of the solution we take 
the situation where H 2 {<x>) is strictly monotonically decreasing in the 
Nyquist band. This implies that there exists only one frequency a><> 
for which KH 2 (u) = X and consequently we get 

* - ms- (29) 

This gives one relation between the unknowns n and o> and another is 
obtained from the power constraint. Since the optimum filter char- 
acteristic is zero when o> > ai , the signal-to-noise ratio is 



'-'/"[aib- <<■■' ! 



J. 

# 2 (wo) # 2 (") 
and an explicit formula for the mean-square is 



T MSE 1 T f ua T 
— In — 2~ = — / m H 2 (w)dc0 wo In H 2 (co ). 

|_ (J a T JO 7T 



(31) 



We now briefly summarize how these optimized formulas are to 
be used : 

(i) For a given transmitted average signal-to-noise ratio p, solve 

eq. (30) for co . 
(u) If wo < t/T, use formula (31) to compute MSE. 
(in) If wo ^ tt/T, use formula (2G) to compute MSE. 

In Section VI we shall illustrate numerically the use of these 
formulas. 

V. SIGNALING FASTER THAN THE NYQUIST RATE 

Here we examine the behavior of the optimized mean-square error 
when the frequency support of an ideal unity-gain channel is smaller 
than the Nyquist rate \T . After deriving the optimized MSE for this 
situation, the possibility of further optimization relative to the signal- 
ing rate when the information rate per unit bandwidth is held fixed 
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will be investigated. This has been an open question thus far and the 
issue is whether increasing the signaling rate beyond the Nyquist rate 
while decreasing the number of levels to maintain a fixed information 
rate is ever beneficial. 

Consider a channel having the characteristic 

where the sets E\ and E 2 form a partition on the frequency interval 
E = {w:0 ^ o) ^ t/T}. By frequency support we mean the measure 
of the set E 2 denoted by m(E 2 ). 

For this channel it is easy to calculate explicitly the mean-square 
error. Observe from eq. (22) that for a piecewise constant channel the 
optimum transmitting filter is a constant when v £ E 2 and zero other- 
wise. The minimum MSE is then calculated from (23) 

- In T ^P 1 = £■ f" T In IK&H*(<*) + l]d» 

= - f In [tfS 2 + l]dco 

7T J u £E t 

= -m(£ 2 )ln[KS 2 + 1], (33) 

IT 

where S 2 is a constant to be determined from the power constraint 
P - -4 /" S2d » = -4<SM#2). (34) 

From this equation and the definition of K = 1/(NqT) it can be 
checked that 

_ Average signal power 

Average noise power in a band = m{E 2 ) 

Substituting this into (33) gives the simple desired formula 

MSE = al{\ + p)-», (35) 

where 

m(E 2 )T 
a = 

7T 

Channel bandwidth ^ . 
2 X Signaling rate 

For fixed p, MSE -> a\ whena = and, as expected, MSE -» o-^l + p)" 1 
when a = 1. It is curious that as long as a ^ 0, MSE — > as p — > ■». 
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When the set E 2 is the interval / = {«:0 ^ a S */To < v/T\, it 

is referred to as being less than the Nyquist band. Since there is no 
mathematical reason for making this distinction we shall refer to 
"signaling faster than the Nyquist rate" whenever a < 1. 

We now investigate whether it is ever advantageous to signal 
faster than the Nyquist rate. Clearly, for fixed a\ = (L 2 - l)/3, or a 
fixed number of levels, (35) shows that MSE degrades rapidly with 
decreasing a. An interesting question, first raised by R. W. Lucky, 7 is 
the possibility of trading L with a to further minimize the mean- 
square error. This is the problem we address. 

Let the source information rate be R = \og 2 L/T bits/second. 
The available bandwidth is equal to 1/ (2Tr)m{Ei) cycles/second. Thus 
the normalized information rate is 

2i t °its 
= a l0gjL ^li' (36) 

Writing (35) in terms of this quantity gives 

M8E(a) = 2 " ~ * (1 I f y (37) 

Letting C = log 2 (l + p) be the ultimate attainable rate according to 
Shannon's theory, eq. (37) can be put into the form 

MSE(a) - (2-"< c - E > - 2-° c )\. (38) 

Note that, since L = 2 aC/2 and the minimum allowable L is equal to 2, 
the parameter a must be in the range {2/8, 1). 

The problem initially posed can now be stated as follows. Find a 
set of a's, 2/8 Si a ^j 1, which minimize eq. (38). We begin by setting 
the derivative of (38) to zero, 

^M = - {C - D*-<-« + C2— = 0, 

from which we find a unique stationary point 

Since MSE(O) = and MSE (a) > 0, the value of a found above 
must be a point where MSE attains a maximum. If this maximum is in 
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the range (0 ^ a ^ 2/8), then the minimum value of MSE is attained 
at the boundary (a = 1). The condition for this to be the case is 
determined from (39). 

log, [^J* SI. (40) 

From this we deduce that as long as 8 < f C, a. = 1 minimizes MSE. 
For this region of 6 and C there is no advantage gained by signaling 
faster than the Nyquist rate. Since C is channel capacity, 8 C = IC 
seems to be a critical rate. If the rate is greater than this critical rate, 
the maximum point lies within the allowable range of a(2/ 8, 1) thus 
raising the possibility that either a = 2/ 8 or a = 1 is a minimum point. 
Suppose a = 2/8 is a minimum point. Equation (38) gives for this 
case 

MSE (a = | ) = 2- 2C '* ^ 2- 8 ' 3 « 0.157. (41) 

Thus we have found a region for which signaling faster than the Ny- 
quist rate appears to be beneficial. But at this high level of MSE, we 
are no longer justified in assuming that the feedback decisions are 
correct most of the time. In fact, what is more likely to happen is 
that errors begin to occur resulting in a larger value of MSE than 
predicted by the error-free model. We are therefore led to the con- 
clusion that a minimum point other than at a = 1 will render an MSE 
to be outside the range of practical utility. To emphasize this point 
further substitute (41) into (38) to get 

MSE(a = 1) = [MSE(2/«S)] E/2 ^^- (42) 

Thus a = 1/ 8 is a minimum point whenever 

[MSE(2/ «)]■'» (^-j=^) > MSE(2/S) 
or 

MSE (2/8) > (gTITl) 2 ' 8 " 2, 

As an example of the use of these inequalities, suppose that 8 = 3 
and C = 3.5 ; therefore, a = f is the minimum point and the achievable 
MSE = 0.198 which can be obtained with a binary system (L = 2). 
On the other hand, M(a = 1) = 2- 8 - 5 (7/3) ^0.206 which can be 
achieved with (L = 2 1 5 = 2.8)!! 

It is interesting to see what MSE can be achieved when only a 
linear equalizer is used. Using formula (15) and following the same 
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reasoning as before yields an expression for the optimized mean- 
square error 

(MSE) linettr = <r\ (1 - a l p + i . (43) 

P -r l 

In this case, unlike in the decision feedback case [eq. (35)3, as P ~~* °°j 
MSE — > 1 — a, when a > 0. Thus the mean-square error cannot be 
made vanishingly small as the signal-to-noise ratio increases without 
bound. 

Expressing (43) in terms of the normalized rate 8 we get 

(MBg) UM ,-gfl[ ' (1 ;J 1 +1 ]. (44) 

Again we seek to minimize (44) with respect to a in the range 
(2/8, 1). It can be checked that (44) has at most one stationary point 
in the range = a = 1. Since a = is a minimum point, the minimum 
in the range (2/8, 1) must lie on the boundary. Thus the condition for 
achieving a smaller MSE when signaling faster than the Nyquist rate 

(a < 1) is 

2 E — 1 
p(l - 2/8) + 1 ^ *—^± 

or 

>" , - ig ^'i-V (45) 

Is it possible to find an 8 ^ C, 8 > 2 which satisfies the inequality 
(45) ? A straightforward analysis reveals that the answer is negative. 
In other words, the Nyquist rate is optimum provided the information 
rate is less than channel capacity. 

VI. APPLICATION TO A CABLE CHANNELt 

This section will illustrate the use of the formulas developed in 
previous sections in a particular application. For this purpose we 
choose a cable channel having frequency characteristic 

H(f) = exp {V-2ia/}. (46) 

We shall develop in detail the applicable formulas, provide asymptotic 
behaviors, and exhibit numerically the relevant parameter trade-offs. 
For comparison purposes, the applicable formulas for the optimum 
linear equalizer will also be developed. 

1 1 am indebted to Dr. Robert Price for calling my attention to Ref. 8 where 
related work is reported. The paper is in Japanese ; however, Dr. Price has an English 
translation. 
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We begin by first considering a suboptimum system where the 
transmitting filter is flat across the Nyquist band and zero outside. 
For this case, the minimum attainable mean-square error is [eq. (23)] 

M 4 = exp -5 j** In l&Kir*^ + l]dwj, (47) 

where 

M d = USE/al, \H(u) | 2 = e-V^ and K = 2a 2 a /TN 

are to be determined from the average power constraint. Since the 
transmitted power P = (o*/T)S*, we find that S*K = 2PT/N = p, 
the transmitted signal-to-noise ratio, where the noise is measured in 
a band = 1/T. Substituting these constants into (47) and making 
some changes of variables result in 



M d = exp I - 2 J* In [pH^ + l]dy 



(48) 



where = a/T. The parameter V2/3 is seen to be proportional to the 
loss of the cable in dB at the Nyquist frequency \T. 

We are interested in the behavior of (48) when p and are varied. 
While it is not possible to express this integral in a closed form, it is 
possible to obtain a rapidly converging series in the two parameters of 
interest from which asymptotic behaviors can be deduced. Appendix B 
shows the details of the development. Different power series apply in 
different regions. The first series applies when In p < V2/3 and the 
second In p ^ V2/3. The results are as follows 

1 : V2/3 ^ In p > 

+j^(-«-"M?-(^^-]- (49) 

2 : In p g V2/3 > 

O , /7 « 1 T «V2/3 "In 

It can be checked that (49) equals (50) when In p - V2J3. The first 
asymptotic behavior is deduced from (50) when p — ► °o and \2/3 is 
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held fixed. For this case we get 

M d ~- . (51) 

P 

Another asymptotic behavior is deduced from (49) as /3 — + °° while 
p is held fixed. In this case 

M d ~ e-o^ie, (52) 

where 

g(p) = (lnp)»yin P + £ ( _ 1) ^ll 

O n=l Up 

In order to make possible performance comparisons, we develop 
similar formulas and asymptotes for a system employing only a linear 
equalizer. The minimum mean-square error applicable in this situa- 
tion is obtained from eq. (15). After substituting the cable char- 
acteristic we obtain 



M 

where 

Ml = 



= 2 /"* \_pe-^y + \y-Hy, (53) 

Jo 

(MSE) linottr 



Here, as in the decision feedback case, rapidly converging series can 
be developed from which asymptotic formulas are deduced. The 
detailed calculations are also given in Appendix B. The desired results 
are 

1 : V2/9 ^ In p > 

Ml=1 _ 30^+^ + ^ In [1 + pe-tf*] 

(54) 



+ L4 ( - l) " +1 ^ 




2 : In p > V2/3 > 


*>-$*[""] 







+S&<-»~*[M?)1- 



(55) 



The asymptotic formulas are readily deduced from (54) and (55). 
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When p — ► °o and is fixed we get 

M L ~ Q(/3) -, (56) 

P 

where 

Q(/S) =■ •** 

On the other hand, when — > °° and p is kept fixed, 




where 



3(lnp)* + x* D (57) 

Ml 1 w — + J' ( ' 



^ = | 1 ( - 1)n+1 ^" 



At this point it is possible to make some judicious performance 
comparisons between the two schemes. The first observation is that, 
in order to get a small mean-square error, p must be large and greater 
than e^. In this case, (51) and (56) apply. On the other hand, when 
V2/3 > In p, and /3 — ► °° , performance deteriorates rapidly as can be 
seen from (52) and (57). Suppose now that a large signal-to-noise 
ratio is available and we wish to obtain the same mean-square error 
in both schemes. How do the signaling rates compare? 

Equating (51) and (56) shows that /3 d /0L ~ 9/4 when these quanti- 
ties are large. In other words, asymptotically, the signaling speed of 
the cable may be increased by more than a factor of two with the 
use of decision feedback. Clearly when /3 is small no significant ad- 
vantage can be obtained from using decision feedback equalization. 

To exhibit these phenomena further, we have used numerical 
integration to evaluate (49) and (53) and checked the accuracy by 
summing terms in the various power series. The results of these 
calculations are exhibited graphically in Figs. 5 through 9. A striking 
feature in all these curves is the manner MSE degrades as /3 increases. 
The linear MSE exhibits a sharp threshold while the MSE for the 
decision feedback equalizer degrades much more gracefully. 

Next we wish to examine the possible payoffs when the inband 
characteristics of the transmitter filter are optimized. To do this 
explicitly, we follow the procedure outlined in Section IV. Equation 
(30) must first be evaluated for the cable characteristics. (We omit 
all straightforward integrations and algebraic manipulations.) Re- 
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Fig. 5— MSE in dB vs V20 for p = 20 dB. 
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writing eq. (30), 



p t [<»> r i i_ ] 

m \ tt y L# 2 (wo) # 2 (*) J 



(**) 



/3L C /3o J' 



(58) 



where 



and 



0o = a/ T , coo = tt/To, = a/T, 

F(s) = e^{4x — 1 ) + l ~ z/2 when .t is small. 
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Fig. &— MSE in dB vs V2/3 for P = 40 dB. 

The explicit evaluation of MSE is as follows : For a given p, solve 
(58) for ft. If /?o > 0, calculate MSE from eq. (26), 

_ i n T MS? j = In Jlf , = In M + ? /^ In ff»(«)d«. 
An explicit evaluation gives 



M„ = 



>l\/20 






/3o>/3. 



(59) 



On the other hand, if (58) yields a O < 0, use formula (31) to compute 
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Fig. 7— MSE in dB vs V2/3 for P = 60 dB. 

MSE. The explicit evaluation for this case gives 
M d = e-K/Jo/^Vi^ /3o > /3j 

1 



p|- + F(2 J 8o)//3o 



K/30//3) 



(60) 



where e^ 2flo was obtained from (58). It can be checked that when 
00 = (J in (58), eq. (59) equals (60) as it must. 

Let us now pause and examine what these optimized results are 
telling us. Suppose /3 is fixed in (58) and p is allowed to increase. 
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Fig. 8— MSE in dB vs V2/3 f or p - 80 dB. 

Eventually a /3 will be found which satisfies (58) and which ultimately 
will be greater than 0. The physical implication of finding a /3 which 
is less than is that the transmitting filter cuts off before the Nyquist 
frequency \T. This will occur only when p is relatively small and thus 
results in a poor MSE. Practically, the region of interest is when p 
is large such that /3 ^ /3, in which case the filter cuts off at the Nyquist 
frequency. In this region (59) applies and, upon comparing (59) with 
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the asymptotic suboptimized result, (51) shows that 



,lV20 



,\jw 



P + 



F{20) 



Since min^ [F(/3) //?] = 1, the optimized result appears to be 
asymptotically equal to the suboptimized result. In other words, in 
the region where In p > V2/3 and p — * °o no benefits are obtained from 
inband optimization. This is also evident from eq. (25) since when In p 
is large relative to V2/3 the structure of the optimum transmitting 
filter is a constant. The situation where /3 < /3 is slightly more com- 
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plicated to compare. Here inband optimization should perhaps be 
beneficial. However, comparisons in this case between the optimized 
MSE and the suboptimized ones must be made on the basis of the 
same transmitted power rather than signal-to-noise ratio because the 
systems operate over different bandwidths. 

Again for comparison purposes, we summarize the formulas that 
apply when the inband characteristics of the transmitting filter in a 
system using only linear equalization are optimized. Berger and Tufts 6 
carried out such an optimization and the procedure is similar to the 
one carried out in Section V. Adopting our notation and the same 
definition of parameters as above we can obtain explicitly the follow- 
ing formulas applicable for a linear system. 

Choose a p and a and solve for O in the equation below : 



p l Po Po 



(61) 



If 00 > p", calculate 



[JW2)] 

P + 
If p* in (61) is < /3, calculate 



M > - F(W ' * > * (62) 



M L = 1 - & + | FW2)e-<^\ (63) 

It is now possible to cross plot the formulas derived in this section ad 
nauseum. We shall show only two sets of graphs. Figures 10 and 11 show 
four curves of MSE/<r* in dB vs V20 where E = 2aP/N and P is the 
transmitted power divided by the parameter N /2a. In each case we 
plot the optimized results and the suboptimized results. The optimized 
decision feedback equalizer results were evaluated from equations 
(58), (59), and (60) and from equations (61), (62), and (63) for the 
linear equalizer. The nonoptimized results are given in (48) and (53) 
respectively. In all cases E = p/0 in dB. Marked on the curves is the 
value of V2p\) where the transmitting filters cut off. We show two 
cases, 10 logio# = 60 and 10 log 10 -E = 100. It appears that inband 
optimization does not provide a great deal of performance enhance- 
ment. As expected, inband optimization yields more improvement in 
the linear equalization scheme than in decision feedback. 

We have also evaluated the optimized results as a function of the 
actual signal-to-noise ratio p, where the noise is measured in whatever 
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Fig. 10— MSE in dB vs V2/3 for E = 60 dB. 
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Fig. 11— MSE in dB vs V2/3 for E = 100 dB. 
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band happens to be optimum. We found insignificant differences 
between these and the sub optimized results shown in Figs. 5 through 

10. 

In concluding this section we wish to stress that in practice error 
propagation problems may negate the indicated theoretical results for 
this channel. When the MSE is large, errors will result. In addition, the 
tap gains of the feedback filter may become quite large causing those 
errors which do result to propagate. 
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APPENDIX A 

Solution of the Wiener-Hopf Equations 

We wish to solve the set of linear equations 

R k = £ M n . k S n , k = 0, - 1, - 2- • -, (64) 



n— o 



where {R k \ -« andilf n _* = tf n _ fc + NoK-k (««-* = 1, n = k;8 n - k = 0, 
n 5* k) are given. 

Since \M n )1 K is a correlation sequence with positive Fourier 
coefficients it is well known that it can be represented as the discrete 
convolution of a sequence {M~\°- n and a sequence {M+}£, namely 

I, = £ M^Mn-t for all n. (65) 

Let the sequence {X n \ "«, be determined from 

Rk= £ MfXk-,; all k. (66) 

M 

Substituting (65) and (66) into (64) gives 

t Mt \x k -t - £ SrMn-t-,} = 0. (67) 

y=o I it--* J 
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Clearly a solution of 

X k = £ SnMn-t, k^O, 
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(68) 



is also a solution of (67). 

Define the two-sided discrete Fourier transform of a sequence j X n ] 1 , 
by 

X(d) = £ X n e inB . 



Take the transform of both sides of (66) to obtain 

R{0) = M + (d)X(e). 

The one-sided transform I £ 1 of (68) is 

X-(6) = S-(d)M-(d) 
and X~(6) is obtained from (69) as 



(69) 



(70) 



(71) 



where [•]- stands for "projection to negative integers only." To 
obtain the projection, expand [_•"] in a two-sided Fourier series and 
retain only the part of the series containing negative/positive coeffi- 
cients (including zero). 

Thus the desired solution is 



s-(e) = ^ 



R(e) 



M-(e) . M+(6) 



(72) 



To proceed further, observe that since M(d) = M + (6)M~(6) and 
M(6) = R(6) + No it is possible to calculate explicitly 



R(0) 

M + (d) 



= M-(9) + 



To' 



where 70 is the dc coefficient of M + (0). 
The final solution for S~(6) is therefore 



S-(B) = 1 - — 



N' 



M-(6)y - 



(73) 



(74) 



As pointed out in the text, there are various methods available for 
calculating ilf ± (^) from a known function M{6). We briefly outline 
one such approach. Since M(0) > 0, ^ ^ 2tt, In M(d) may be 
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expanded in a two-sided Fourier series 

In M(9) = E 7ne ine + E yU™. 
Knowing the sequence {y£\!«, we can get immediately 
M+(6) = exp 



E yU M 

;i=0 



(75) 



(76) 



and 



M-(0) = exp E 7,7e''" 6 
{ » = -» 

Notice that the dc term of M+(6) equals the dc term of M~{6). 

APPENDIX B 

Evaluation of Integrals 
B.l Decision Feedback 

The detailed evaluation of 



/ = 2 f In [1 + pe-^yty 
Jo 



(77) 



is accomplished as follows: Change the variable of integration to 
x = (V4/fy — lnp) which gives 



J r-ftji-ln 



(78) 



1 rytp— m p 

I = - a I ln[l + e-'Jx + In P yix 

P J -lnp 

Assume V2/3 > In p > and write (78) as 

r -KL. + /. ) 

= 1 P" a; (In p - a;)rf.T + i /"' ' (In p - .r) In [1 + <r*]<te 

P in P Jo 

1 /■ VS— 1 n P 

+ i / (a; + In p) In [1 + e~*y,x (79) 

P Jo 

_<ln,)' dnp) !+ l p • ) ,„ [i + r.]*, 



2/5 



+ 



1 r 

P Jo 



+ In p) In (1 + er*)dx. 
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Since x > in the range of integration, expand 

ln[l +e-*] = L (- l) n+1 — 

71 =1 ** 

and substitute into (79) to obtain 

B n (^- lnp) lnp A„(V2/3-lnp) 1 

n (8 n I 

where 

A„(g) - i (1 - <r««) 

n 

and 

__ 1 - e~"* - nfrr"£ 
*«(*) ^ • 

Collecting terms and recognizing that 

o-> 1 ') 

A K } n> 12 
we finally get 

. _ (In p)» + tMnp |2f ,_ n „ + ,(p^i" 

60 \0«ti* 1; n 2 

+ L? 1 ( - l)n+l Mp^- (pe_ ^ )n ]- (81) 

When In p > V2/3 > 0, (77) can be expressed in the form 
I = I f (a + In p) In [1 + e-]da; 

P ./-lnp 

1 /■- (lnp-VW) 

+ 5 / (a + In p) In [1 + e~^dx 

P Jo 

= \ /""' (lnp - x)\_x + ln(l + e~*)ytx 
P Jo 

-I l U '* (In p - z)[.r + In (1 + er'ftdx. (82) 
p yo 

At this stage In ( 1 + e~ x ) is again expanded in a power series and when 
the terms are collected we obtain 

/ = lnp-?V2^ + ^^ ( _ 1)n+ie __ 

+ J £ (- D n+1 4n CI - e"^]- (83) 

p „ = 1 71 p 
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B.2 Linear Equation 

The integral we wish to evaluate here is 

/ = 2 /"* (1 + pe-*M)- l dy. (84) 

Jo 

We follow the identical procedure as in the previous case. First change 
the variable of integration to obtain 

i r*^'(* + h»W (85) 

Assume that V2/3 > In p > and write 

I= AL. + L ) 

1 /-VS-inp (a; + lnp) , . 1 /"'""e-^lnp - x) , 

= ,3/0 T + F=-* + *7. i+«- d * 

= 1 (V20-lnp)(V20 + lnp) 
2/3 

+ i t (- l)"[5n(V2^ - lnp) + Bn(lnp)] 

P n = l 



+ T f (- l)»[A n (V2^-lnp) -A«(lnp)] 



In p 

■ ! 

\2 _2 



-1-^-5+ - ■"" ! " ' 



+l±i-v +i h(? + ^* i) ")- (86) 

When In p > V2/3 > we get 

r 1 f ln "e- x (\n P - x) 1 /-inp-^ e-^lnp - x) , 

1 ~ Jo 1+6- ** ~ i§ Jo 1+e- ^ 



- I t (- l)" +1 [lnpA n (lnp) - .B„(lnp) 

P n = l 



+ In pA„(ln p - V20) + B«(ln p - V20)} (87) 
and after collecting terms we finally obtain 

^K-?H5<-»"i[M?r]' 



/ = 
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